Detecting Pedestrians and Cars in 3D

In this report, we address the task of detecting pedestrians and cars and further localizing them in 3D. This is a challenging task owing to variable appearance that humans can adopt and different viewpoints for cars. Pedestrian and car detection has been well studied over the years. A number of robust feature sets [8],[3], [9] have been proposed which successfully discriminates the human form from cluttered background and even making the detection task illumination invariant. We choose one such feature set, locally normalized Histogram of Oriented Gradient (HOG) in order to implement our pedestrian and car detection task. Once the module for person and car detection is constructed, it can be subsequently fused to get the 3D detection in the following way. We use a single video stream from a calibrated stereo rig and then compute the camera pose and ground plane estimate for each image of the video stream. This can be achieved through standard Structure from Motion technique. The ground plane is then estimated by fitting a plane to the 3D points available from the horizontal sweeping of the scene. Taking help of known camera pose, 2D detections would then be converted to 3D observations, similar to Leibe et.al [4]. The input data for Leibe et.al. [4] are two video streams from a calibrated stereo rig. Car and pedestrian detectors are applied to both the camera images and then using the estimated camera pose these detections are converted to 3D observation by combining responses of multiple detectors if they refer to the same object in the scene. In this project, we just use a single video stream and its corresponding depth map as input data. In brief, we perform the following three steps: (1) Using standard car and pedestrian detector on the video frames to get the bounding boxes (2) Estimating a reasonable ground plane from the depth map data for mapping the bounding boxes in 2D to 3D and (3) Correcting for the orientation of the 3D boxes. We demonstrate this method on the Urbanscape Video stream, the depth map data for which are available.